Croakwood Devlog #12 - A Tale Of Optimization

We've recently started working on Mac support for Croakwood. While there's still some work left to do it's generally working now, but the performance initially wasn't quite as good as we were hoping. So we investigated what's going on and here's the story of what we found :)

When rendering a game, the CPU needs to tell the GPU what it's supposed to draw, so it needs to send a long list of instructions to the graphics card that say which material you want to use for drawing and what mesh you want to draw. So it's something roughly like this: "I want to draw something that looks like stone; I want to draw the round_stone mesh; I want to draw something that looks like stone; I want to draw another round_stone mesh somewhere else; I want to draw something that looks like stone; I want to draw the angular_stone mesh; I want to draw something that looks like wood; etc".

Having to do this for every single object would be quite a long list. Putting together this list, submitting it to the GPU and having the GPU work through it takes some time.
As an optimization Unity has something called "instancing". With this, instead of submitting every object you want to draw individually, it submits a list of positions for each mesh you want to draw. This allows to shorten the instruction list to "I want to draw something that looks like stone; I want to draw the round_stone mesh in the following locations: A, B, C; I want to draw something that looks like stone; I want to draw the angular_stone mesh in the following locations: D, E, F; I want to draw something that looks like wood; ...".

This is quite a bit faster than submitting each object individually, but if you want to draw many different meshes it's still adds up to quite a lot. Creating and submitting all these render instructions was one of the biggest performance costs for Parkitect, so speeding this up was a big priority for us for Croakwood.

On "modern" systems (pretty much anything produced in the last ~10 years or ~5 years for Mac) there's support for what's called "multi-draw", which allows us to submit a list of all the different meshes we want to draw with a certain material in one go. This shortens the instruction list to something like "I want to draw something that looks like stone; I want to draw the round_stone mesh in the following locations: A, B, C; I want to draw the angular_stone mesh in the following locations: D, E, F; I want to draw something that looks like wood; ...".

Luckily Unity just added support for multi-draw when we started working on Croakwood. Weirdly they didn't simply automatically use it wherever possible, but they added a method where you can manually say what you want to render and then according to the documentation it would use multi-draw on modern systems and fall back to instancing on older systems. That was perfect for us and so we used it.

When we were looking into why the Mac version wasn't performing that well we noticed that the rendering was the main issue. Rendering performance isn't the greatest even on modern Macs so that wasn't all too surprising, but it was still slower than what we expected. We were testing on a M1 Mac Mini that should have support for multi-draw, but we noticed that it wasn't using multi-draw, it was using instancing.
We checked the Windows version of the game on a modern system... and it wasn't using multi-draw either.
Weird! Clearly we must have been doing something wrong, but we couldn't figure out what it was.

We headed to the Unity forums for help and found a thread where other people had noticed this problem as well and had reported it to Unity, and as it turns out... the documentation was wrong. There's no support for multi-draw in Unity at all. Oof :/

Looking around for solutions, someone had built a plugin to add support for multi-draw to Unity, which is amazing!
We didn't want to be dependent on a plugin that might break in newer Unity versions though and ultimately came up with a solution ourselves that isn't true multi-draw but should have fairly comparable performance, at least on the CPU side.
This got the game to run at about 58 FPS instead of 30 on the M1 Mac Mini.
Still not quite perfect and there's some more work to do, but a huge improvement :)

There's surely good reasons for why Unity aren't supporting multi-draw yet but it is a bit sad since it seems like a relatively easy way to significantly improve performance in many games.
Hopefully they add it in the future, and then we should be able to switch to it easily. Until then our solution gets the job done and we can move on to other tasks.

Our multi-draw alternative

If you're interested in the technical details:

We're using one call to RenderPrimitivesIndirect per material that we want to render. We have a compute shader that does frustum culling and LOD selection (Unity does this on the CPU, which also costs a lot of time. We need the CPU for our town simulation though). It sums up the vertex count of all visible meshes for the given material.

The difficulty is that once we get to the vertex shader we only get a vertex index but don't have any information to which mesh it belongs.

To solve this, our compute shader also builds a mapping table that says which mesh a vertex belongs to. Obviously it would be quite wasteful to have one entry in this table per vertex, so our mapping table only contains one entry covering a certain amount of vertices at once. We ended up choosing one entry per 384 vertices. This means there is a bit of waste because the total sum of vertices we need per material is of course not always divisible by 384, so we are discarding some unnecessary vertices in the vertex shader.
This isn't ideal, but the amount of vertices we discard isn't super high in the end, and doing this is still much faster than simple instancing.

Debug view for the mapping table, with vertices sharing an entry being drawn in the same color

Croakwood - Early Access Release Window & Gameplay Trailer

We're on track to release Croakwood into Early Access this winter!
Check out our new trailer that's giving a look at lots of new things:

What is Early Access?

Early Access means the game is available to buy and play while it's still being actively developed. The game regularly receives free updates with the newest completed features and content.

Why Early Access?

We think that we are able to make a better game by working together with our community. We experienced this with Parkitect, where the feedback we received has been incredibly helpful to make a bigger and better game than we would have been able to do on our own. While Croakwood has the benefit of so many of the lessons that were gained through that process, it is a very different game that both leans on our strengths and pushes into new territory for our small, but mighty team.

We have been developing this game for many years according to our own ideas, and now we are excited to see what people enjoy and what might need to be improved or expanded.
Early Access allows players to share their experiences and ideas about what the game should be, and gives us an opportunity to incorporate that feedback and adapt our plans accordingly with more flexibility than may be possible past full release.

Approximately how long will this game be in Early Access?

We expect it's going to take about 1-2 years to add the content and features that we have planned for the full release but we'll adjust our plans based on the feedback we receive and what the game needs, so this may change.

How is the full version planned to differ from the Early Access version?

We plan to add more content and features: more decoration and furniture sets, resources, workshops with new resource production chains, world biomes, activities for the villagers, characters you can meet, tasks for you to do and additional depth for the town management gameplay.

What will be the state of the Early Access version?

You will be able to construct and run a nice little town: design and decorate houses with many available decoration and furniture items with many customization options; build up various resource production chains and watch the villagers gather, craft and transport resources; watch enjoyable villagers as they inhabit your town; fulfill requests and meet your villagers needs.

Everything we've shown somewhere (like in devlog posts, trailers, screenshots or in the store page description) will be available.

Will the game be priced differently during and after Early Access?

The Early Access price will be lower than we expect it to be for the full release. We think it’s the fairest approach if the price reflects the state of the game at any given point in time, so we'll gradually increase the price as more content gets added.

How are you planning on involving the Community in your development process?

We’re reading and listening to all the feedback we receive on Steam forums, on our Discord, via our website and on social media.
We’re regularly posting updates on the development progress to allow everyone to track how things are going.

When exactly is the Early Access version going to be available?

We don't have a date yet! But we're confident that we can get everything we want to get done ready in time for an Early Access launch approximately within the November-February timeframe. As soon as we know more we'll update you :)

As you may be aware game development time requirements can be notoriously hard to estimate correctly and unexpected life events can happen. We're not expecting any delays, but in case anything unexpected happens we'll let you know.

Croakwood Devlog #11 - World Design

Hey! Don't forget to check out Wholesome Direct this Saturday :)

Wholesome Direct

1780761600

This month we made a lot of progress in how our game world is put together and designed.

The way our game world works is a bit like a puzzle. We have individual pieces that we design by hand, and then the game automatically pieces them together to create the world. We can rotate pieces, randomize where the connection points are and add some random content into the pieces for some additional variety. This way we can create a unique world for every player while still having a lot of manual control over the design of the pieces to make them look beautiful and interesting.

Here's what our editor for these pieces looks like:

We can add terrain to a piece, define where the playable area and possible connections to other pieces are and add decorations.
This isn't a tool that's intended for players currently, but eventually it would be really cool to make it available for everyone and to allow sharing player-created map pieces.

For gameplay, the idea is that players don't have access to the entire world from the start. You start in a certain location and then gradually get access to additional areas.

When we initially worked on this system a few years ago we thought it would be the best idea to pre-generate the entire world when you start a new save file and then mark everything outside the starting area "unexplored" - so it exists, you just can't access it yet. We thought generating the whole world at once would make things a bit easier for us, like controlling how the game progresses (like for example putting more advanced resource types further away from your starting area).
Here's a screenshot from a tool we used to prototype this. The colored boxes show which parts of the terrain come from the same puzzle piece, the light grey background shows the playable area and the darker grey shows the border around the playable area where we generate the forest.
Short green lines mark the connections between pieces and there's also some generated rivers in blue.

There's some problems with pre-generating the entire world though:

  • it's kind of wasteful to generate stuff that the player hasn't unlocked yet (takes up space in the savegame file and might require loading more data than necessary)
  • if we add more pieces to the game, players have to create a completely new save file and restart their town to have a chance to encounter them

We changed that this month. Now pieces only get added to the world once you unlock a new area.
This has been on our task list for a long time, so it feels really good to finally have it done!

And we started replacing our placeholder test pieces with properly designed ones.
It makes a huge difference for how alive the world feels and instantly makes everything look a lot more interesting and nicer, so that is a lot of fun for us :)
It's very exciting to see how every single piece being added immensely increases the variety of worlds being generated!

Croakwood Devlog #10 - How villagers interact with the world

Hey, it's been some time!
If you want to know what we've been up to, make sure to check out Wholesome Direct on June 6th :)

Wholesome Direct

1780761600

In one of the previous posts I described how the villagers navigate around the world and in this one I want to talk a bit about how they decide where to go and what to do.

Let's take this woodcutter hut for example:

There's a tool rack containing an axe, a few trees around the hut and a place where logs can be stored.
It's quite obvious what needs to happen here in order to produce logs, but how do we put this into program code?
An easy solution would be having some code that says "if you're a woodcutter, here's step by step what you need to do: get the axe, chop a tree, carry the log to storage and put away the axe".
This could be done quite easily with a state machine or if you want something a bit more powerful with a behaviour tree.

That's how we implemented it initially and what we also did for the guests in Parkitect.
However, there's a few problems:

  • What if the woodcutter gets interrupted after chopping the tree? For example the player might have built a wall that prevents the woodcutter from returning to the workshop. The woodcutter is still holding the axe, and now there's logs on the floor that need to be transported away. Suddenly our clear list of instructions for what to do doesn't work anymore. We could provide multiple lists of instructions for different situations or make the instructions a bit more flexible (the first instruction in the example could say "get the axe if you are not already carrying it")... but that means we need to think of every possible combination of things that could possibly happen.
  • This doesn't create very natural looking behaviors. What if the woodcutter decides to chop two trees in a row? They would put away the axe after chopping the first tree, just to immediately pick up the axe again for the second tree. Maybe we could remove the "put away axe" instruction and allow them to keep carrying the axe, but when do they put it away then? For example they should definitely not hold the axe anymore when they go sleep. We could have a "put away any tool you're holding" instruction as the first item in our "how to sleep" instruction list, but again: for every behavior a villager can do we'd have to think of every possible state they might be in and make sure it's something valid.

So this felt like a pretty fragile solution and we've been struggling with it for some time until we learned about Goal-Oriented Action Planning (GOAP).
GOAP is a planning system that generates the necessary list of instructions in order to achieve a certain goal.
It's a bit like pathfinding, but instead of searching through a list of positions to find a path to a certain location you're searching through a list of possible instructions to find a valid sequence.
The way it works is that first we need to define what actions are available to the villager, and every action also has a list of preconditions that need to be fulfilled in order to be able to execute the action, and a list of effects the action has.
For our woodcutter example it looks like this:

If we now give the villager a goal of "has put down log" the GOAP algorithm takes all of these actions and tries out all valid combinations in a somewhat smart way until it finds a sequence where the last action has an effect of "has put down log".
So for example the trees now offer a "chop" action that has a precondition of "has axe" and an effect of "has log". The villager is not holding an axe currently so this action is not possible.
The axe tool rack has a "pick up axe" action with a precondition of "has empty hands" and an effect of "has axe". The woodcutter is not holding anything currently so this action is possible, and then afterwards the "chop" action becomes possible because its precondition has been fulfilled now and it keeps going like this until finding the "put down log" action of the log storage.

This is really cool, because now we just have to define all of the individual actions correctly which is a lot more manageable and less error-prone.
What's also great is that this makes adding new objects and behaviors into the game quite easy! We just have to set up the new actions an object provides and maybe define a new goal and the villagers can immediately interact with it. It feels so much easier than our initial approach that it's a bit like magic.
Sometimes it even leads to surprises where the villagers reach a goal in a way that makes complete sense but we did not expect :)

The downside of course is that generating the action sequences is not free in terms of computations, especially if there's a large list of possible actions to try, so it's important to keep that list as small as possible. What's also a bit tricky is that Croakwood towns can have many villagers and they are all following their own plans, but because it usually takes quite some time to complete an action list there's not many of them having to create new plans at the same time.
So far this has not been an issue but it's something we'll need to keep an eye on.